Empirical Comparison of Incremental Learning Strategies for Genetic Programming-Based Keep-Away Soccer Agents

We consider the problem of incremental transfer of behaviors in a multi-agent learning test bed (keep-away soccer) consisting of homogeneous agents (keepers). One method for this incremental transfer is called the easy missions approach, and seeks to synthesize solutions for complex tasks from those for simpler ones. In genetic programming (GP), this has been achieved by identifying goals and fitness functions for subproblems of the overall problem. Solutions evolved for these subproblems are then reused to speed up learning, either as automatically defined functions (ADFs) or by seeding a new GP population. Previous positive results using both approaches for learning in multi-agent systems (MAS) showed that incremental reuse using easy missions achieves comparable or better overall fitness than monolithic simple GP. A key unresolved issue dealt with hybrid reuse using ADF plus easy missions. Results in the keep-away soccer domain (a test bed for MAS learning) were also inconclusive on whether compactness-inducing reuse helped or hurt overall agent performance. In this paper, we compare monolithic (simple GP and GP with ADFs) and easy missions reuse to two types of GP learning systems with incremental reuse: GP/ADF hybrids with easy missions and single-mission incremental ADFs. As hypothesized, pure easy missions reuse achieves results competitive with the best hybrid approaches in this domain. We interpret this finding and suggest a theoretical approach to characterizing incremental reuse and code growth.